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EXECUTIVE  SUMMARY 


The  safe  and  reliable  operation  of  software  within  civil  aviation  systems  and  equipment  has 
historically  been  assured  through  the  application  of  rigorous  design  assurance  applied  during  the 
software  development  process,  hicreasingly,  manufacturers  are  seeking  ways  to  use  software 
that  has  been  previously  developed  for  other  domains,  has  been  previously  certified  for  use  in 
lower  criticality  aviation  applications,  or  has  been  certified  to  earlier  versions  or  different 
standards  than  those  currently  employed.  Product  service  history  is  one  method  for 
demonstrating  that  such  software  is  acceptable  for  use  in  a  new  application.  In  fiieory,  product 
service  history  would  seem  to  be  a  fairly  simple  concept,  both  to  understand  and  to  apply. 
However,  in  practice,  such  use  has  proved  extremely  problematic,  as  questions  of  how  to 
measure  the  historic  performance  and  the  relevance  of  the  provided  data  have  siufaced.  To  date, 
no  specific  guidance  has  been  produced  to  aid  in  the  formulation  of  service  history  approaches 
beyond  the  limited  discussion  in  DO-178B,  “Software  Considerations  in  Airborne  Systems  and 
Equipment  Certification.”  This  research  effort  was  designed  to  collect,  analyze,  and  synthesize 
what  is  known  and  understood  about  applying  product  service  history  and  then  to  adapt  this  data 
into  a  handbook. 

This  technical  report  presents  the  results  of  this  research  effort  in  the  form  of  a  handbook 
intended  for  use  by  both  the  Federal  Aviation  Administration  and  industry  in  formulating  and 
evaluating  service  history  arguments.  Using  a  taxonomy  of  questions  derived  from  the  defimtion 
of  product  service  history  in  DO-178B,  both  quantitative  and  qualitative  considerations  are 
explored.  This  discussion  is  extended  by  inclusion  of  additional  questions  from  other  industries 
in  which  service  history  is  used  in  evaluating  software  maturity.  Finally,  a  set  of  worksheets  are 
derived  that  can  be  used  by  anyone  evaluating  the  relevance  and  sufficiency  of  service  history 
data  for  possible  certification  credit.  The  handbook  concludes  with  a  discussion  of  process 
assurance  and  equivalent  levels  of  safety  for  the  purposes  of  determining  when  and  what  type  of 
supplemental  data  may  be  required  to  fulfill  the  objectives  of  DO-178B  in  conjunction  with  the 
use  of  service  history. 
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1.  INTRODUCTION. 


1.1  PURPOSE. 

The  purpose  of  this  handbook  is  to  aid  industry  and  the  Federal  Aviation  Administration  (FAA) 
in  the  application  of  product  service  history  for  certification  credit  within  the  firamework  of 
DO-178B,  “Software  Considerations  in  Airborne  Systems  and  Equipment,”  as  invoked  by  FAA 
guidance,  Advisory  Circular  (AC)  20-1  15B\ 

1.2  SCOPE. 

The  scope  of  this  handbook  covers  the  subject  of  product  service  history  for  software  used  for 
certification  credit  in  approval  of  airborne  systems.  The  content  may  be  useful  in  other  situations 
where  product  service  history  credit  is  being  sought. 

1.3  BACKGROUND. 

During  the  creation  of  DO-178B,  product  service  history  was  identified  as  a  possible  alternative 
method  for  demonstrating  compliance  to  one  or  more  of  the  objectives  in  DO-178B.  To  date, 
use  of  this  method  has  been  limited  due  to  bofti  the  difficulty  m  demonstrating  the  relevance  and 
sufficiency  of  the  product  service  history,  as  well  as  a  lack  of  any  consistent  guidance  for 
approaching  such  a  demonstration. 

This  handbook,  the  result  of  an  FAA  study,  attempts  to  capture  in  one  place  what  is  known  about 
the  topic  of  product  service  history.  Using  the  guidance  provided  in  DO-178B  as  a  starting 
point,  other  safety-critical  industries  were  canvassed  in  an  attempt  to  identify  if  service  history 
was  being  used  as  part  of  system  evaluations,  and  if  so,  in  what  manner.  Similarly,  other  sources 
of  guidance  for  the  aviation  industry  were  explored,  most  notably  the  work  accomplished  by 
RTCA  committees  (SC- 180  and  SC- 190)  and  by  the  Certification  Authorities  Software  Team 
(CAST). 

The  SC- 180  committee  produced  DO-254,  “Design  Assurance  Guidance  for  Airborne  Electronic 
Hardware.”  It  outlines  how  product  service  experience  may  be  used  “to  substantiate  design 
assurance  for  previously  developed  hardware  and  COTS  components.”  DO-254  was  released  in 
April  of  2000.  DO-254’s  treatment  of  product  service  experience  is  contained  in  two  separate 
sections,  11.3  and  Appendix  B,  paragraph  3.2.  The  guidance  in  11.3  closely  parallels  the 
guidance  in  DO-178B  for  product  service  history.  However,  the  guidance  in  the  appendix 
requires  additional  design  assurance  for  levels  A  and  B  hardware  if  service  experience  is 
claimed.  There  is  also  a  requirement  to  link  any  analysis  of  product  service  history  experience 
into  the  functional  failure  path  analysis  for  levels  A  and  B.  This  is  analogous  to  the  tie  back  to 
system  safety  objectives  required  in  12.3.5  of  DO-178B. 


‘  DO-178B  is  published  by  RTCA,  Inc.  and  is  used  widely  in  the  United  States.  It’s  European  counterpart,  ED- 
12B,  is  pubUshed  by  EUROCAE  and  is  used  throughout  EUROPE.  The  documents  are  technically  identical.  The 
Joint  Airworthiness  Authorities  (JAA)  invokes  the  use  of  DO-178B  via  Ten:5)orary  Guidance  Leaflet  (TGL)  No.  4 
in  a  similar  fashion  to  AC  20-1 15B. 
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SC- 190  is  still  an  active  committee,  although  their  final  work  products  are  ciurently  moving 
through  editorial  review  as  of  the  publication  of  this  handbook.  Their  outputs  include  DO-248B 
and  a  guidance  document  for  nonairbome  Communication,  Navigation,  Surveillance/Air  Traffic 
Management  (CNS/ATM)  ground  systems.  Within  DO-248B,  fi'equently  asked  question  No.  19 
and  discussion  paper  no.  4  specifically  address  the  use  of  product  service  history.  The  content  of 
these  items  have  been  reflected  in  this  handbook.  Considerable  discussion  occurred  in  the  SC- 
190  CNS/ATM  subgroup  on  the  establishment  of  a  tiered  approach  to  minimum  service  history 
duration  based  on  criticality  levels.  No  consensus  could  be  reached  on  the  minimum  duration 
since  the  proposals  were  derived  firom  the  software  reliability  field  that,  by  some,  is  not  viewed 
to  be  a  mature  field. 

CAST  is  comprised  of  representatives  of  certification  authorities  from  Emope,  Canada,  and  the 
United  States.  CAST  meets  regularly  to  discuss  technical  and  regulatory  matters  related  to  the 
umform  interpretation  of  DO-178B  and  related  guidance.  CAST  produced  a  position  paper  on 
the  subject  of  product  service  history.  Both  the  final  paper  and  a  number  of  earlier  drafts  were 
considered  in  the  course  of  completing  this  research  effort. 

In  addition  to  the  aviation  sources  mentioned  above,  numerous  references  were  reviewed  from 
the  nuclear,  military,  consumer  products,  and  medical  devices  domains,  as  well  as  general 
software  literature.  The  most  mature  treatment  of  the  topics  were  found  in  Europe,  most  notably 
in  standards  published  by  the  United  Kingdom  Ministry  of  Defense  (MoD)  and  by  the  safety¬ 
consulting  firm,  Adelard.  In  addition  to  the  written  materials  that  were  reviewed,  a  series  of 
interviews  were  conducted  with  practitioners  in  these  other  fields  to  fiufher  explore  exactly  how 
the  subject  of  service  history  was  addressed  in  practice. 

The  final  activity  prior  to  the  actual  creation  of  this  handbook  was  the  conduct  of  a  detailed 
breakout  session  during  the  FAA’s  National  Software  Conference  held  in  Boston  in  June  2001. 
Preliminary  results  of  the  study  were  shared  and  feedback  was  sought  related  to  specific  issues 
arising  from  the  product  service  history  definition.  Both  the  interviews  and  the  breakout  session 
served  to  validate  the  findings  from  the  literature  review  and  contributed  greatly  to  the  final 
handbook  contents. 

1.4  RELATED  ACTIVITIES/DOCUMENTS. 

The  following  dociunents  relate  directly  to  the  issues  addressed  herein  and  define  the  nature  of 
the  problem  studied  in  this  evaluation: 

a.  DO-178B/ED-12B 

b.  DO-254/ED-80 

c.  DO-248B/ED-94B 

d.  Federal  Aviation  Regulations  (FARs)  (more  formally  known  as  Title  14  Code  of  Federal 
Regulations  (CFR)  and  associated  ACs,  most  notably 

•  XX.1309 

•  XX.1301 

•  XX.901 

where  XX  represents  the  aircraft  type  for  example,  CFR  Parts  25.1309, 23.1309  etc. 
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In  addition,  FAA  Notices  8110.85  and  8110.89  are  relevant  to  this  discussion.  Finally,  specific 
discussion  papers  presented  at  the  RTCA  SC-167  and  SC-190  meetings  were  reviewed  and 
considered.  In  many  cases,  these  papers  contained  useful  ideas  that  were  not  included  in  the 
final  products  of  their  associated  committees  due  to  a  lack  of  consensus  or  the  constraints  placed 
on  the  committee. 

1.5  DOCUMENT  STRUCTURE. 

This  handbook  is  comprised  of  seven  sections,  and  one  appendix.  A  brief  summary  of  the 
contents  is  provided  here  for  reference. 

•  Section  1  provides  introductory  material  including  the  purpose,  scope,  related  documents, 
background,  document  structure,  and  use  of  the  handbook. 

•  Section  2  discusses  DO-178B’s  treatment  of  the  product  service  history  alternative 
method,  as  well  as  its  definition. 

•  Section  3  provides  a  detailed  discussion  of  the  elements  that  comprise  the  product  service 
history  definition. 

•  Section  4  discusses  the  relationship  of  product  service  history  to  both  process  and  product 
assurance. 

•  Section  5  draws  the  various  aspects  of  a  product  service  history  argument  together  for  the 
purposes  of  illustrating  how  an  equivalent  level  of  safety  argument  might  be  made  using 
product  service  history  data. 

•  Section  6  is  the  sxunmary. 

•  Section  7  contains  a  bibliography  listing  the  most  relevant  sources  of  information  on 
service  history  and  related  topics. 

•  Appendix  A  provides  a  series  of  worksheets  that  may  be  used  to  evaluate  product  service 
history  data. 

Note:  Throughout  this  handbook,  the  language  and  the  philosophy  of  DO-178B  are  retained. 
For  example,  the  vocabulary  used  in  various  domains  of  this  research  is  different  from 
that  used  in  DO-178B.  Words  such  as  “in-service  history,”  “field  data,”  and  “item 
history”  are  used  for  product  service  history.  A  translation  has  been  performed  to 
maintain  commonality  of  terms  with  those  used  in  DO-178B.  Similarly,  the  terms 
product  service  history  and  software  service  history  are  used  interchangeably. 

1.6  USING  THE  HANDBOOK. 

This  handbook  has  been  designed  to  capture  industry’s  best  practices  used  in  evaluating  product 
service  history  for  possible  certification  credit.  Practitioners  are  encouraged  to  review  the 
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commentary  in  sections  1  through  3  when  initially  contemplating  the  use  of  product  service 
history  on  a  project.  The  worksheets  contained  in  appendix  A  of  this  handbook  can  be  used  in 
performing  an  evaluation  using  the  questions  and  ideas  discussed  in  section  3. 

Once  the  initial  evaluation  has  been  completed  using  sections  1-3  and  appendix  A  of  this 
document,  sections  4  and  5  can  be  used  for  ideas  on  how  to  supplement  the  service  history  data  if 
necessary.  The  need  for  such  supplemental  activities  is  a  result  of  the  inclusion  of  12.3  in  DO- 
178B  which  states  that  all  alternative  methods  be  shown  to  meet  the  objectives  of  DO-178B. 
Since  product  service  history  is  often  being  considered  because  complete  development  data  is 
unavailable,  multiple  alternative  methods  may  be  needed  to  satisfy  all  DO-178B  objectives 
(more  on  this  in  section  5).  Any  use  of  service  history  should  be  discussed  in  the  Plan  for 
Software  Aspects  of  Certification  (PSAC)  and  coordinated  with  the  appropriate  Aircraft 
Certification  Office  (ACO). 

Note:  This  handbook  is  the  output  of  a  research  effort.  It  does  not,  by  itself,  constitute  policy  or 
guidance.  The  FAA  may  use  this  handbook  in  the  creation  of  future  policy  or  guidance. 

2.  DO- 178B  FRAMEWORK. 

DO-178B  outlines  a  total  of  66  objectives  that  should  be  satisfied  for  software  with  the  highest 
potential  impact  on  safety  in  the  event  of  its  failure.  Four  additional  levels  of  software  are 
provided,  each  wifii  a  decreasing  number  of  objectives  that  must  be  satisfied  as  the  potential 
safety  impact  is  reduced.  All  of  the  objectives  are  described  in  the  context  of  the  software 
development  process  and  a  series  of  integral  processes  that  cut  across  the  entire  software 
development  effort.  In  addition,  DO-178B  discusses  a  small  number  of  alternative  methods  for 
demonstrating  compliance  to  one  or  more  of  the  66  objectives.  Product  service  history  is  one 
such  alternative  method. 

2.1  THE  DEFINITION  OF  PRODUCT  SERVICE  HISTORY. 

DO-178B  defines  product  service  history  as  “a  contiguous  period  of  time  during  which  the 
software  is  operated  within  a  known  environment,  and  during  which  successive  failures  are 
recorded.”  This  definition  has  three  major  components: 

•  Problem  reporting 

•  Environment 

•  Time 

For  the  pmposes  of  this  handbook,  the  environment  component  has  been  subdiAuded  into  two 
pieces.  The  first  one  is  focused  on  external  operations  such  as  operating  modes,  people  and 
procedures,  and  the  second  is  focused  on  computing  (hardware  environment)  aspects  of  the 
software.  Likewise,  the  problem  reporting  component  has  been  broadened  to  include  all  facets 
of  configuration  management  as  they  relate  to  the  use  of  product  service  history. 
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DO-178B,  Section  12.3.5,  states  that  the  acceptability  of  any  argument  predicated  on  the  use  of 
product  service  history  depends  on  six  items: 

•  configuration  management  of  the  software 

•  effectiveness  of  problem  reporting  activity 

•  stability  and  maturity  of  the  software 

•  relevance  of  product  service  history  environment 

•  actual  error  rates  and  product  service  history 

•  impact  of  modifications 

The  next  section  provides  a  detailed  look  at  the  1 1  guidance  statements  in  Section  12.3.5  of  DO- 
178B  as  they  relate  to  demonstrating  the  above  six  items. 

2.2  ANALYSIS  OF  PRODUCT  SERVICE  fflSTORY  IN  DO-178B. 

Table  1  was  constructed  as  a  result  of  analysis  of  current  guidance  given  in  Section  12.3.5  of 
DO-178B.  Each  item  in  this  section  was  studied  to  understand  what  questions  must  be  asked  to 
get  pertinent  information  and  what  additional  considerations  are  not  discussed  directly  in  DO- 
178B.  The  components  of  time,  environment,  operations,  and  problem  reporting  have  been 
included  to  categorize  each  of  the  guidance  statements  from  DO-178B.  This  taxonomy  will  be 
explored  in  detail  in  section  3. 

Column  1,  DO-178B  Section  12.3.5  reference,  contains  each  of  the  11  guidance  statements 
concerning  product  service  history  as  it  spears  in  DO-178B. 

Column  2,  observations  on  DO-178B  Section  12.3.5,  provides  a  brief  commentary  on  the 
guidance  statement  discussing  how  that  item  may  be  met  and  in  what  way. 

Column  3,  software  service  history  questions,  provides  a  short  list  of  questions  that  can  be 
directly  derived  from  the  DO-178B  statement.  Note  that  these  questions  are  expanded  in  the 
worksheets  found  in  appendix  A  using  best  practices  taken  from  other  domains  that  employ 
service  history. 

Column  4,  question  category,  places  the  DO-178B  guidance  statement  in  one  or  more  of  the  four 
categories  used  throughout  this  handbook  to  discuss  ftie  various  aspects  of  software  service 
history.  These  categories  are  further  explored  in  section  3. 
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2.3  RELATIONSHIP  WITH  PREVIOUSLY  DEVELOPED  SOFTWARE. 


DO-178B  uses  the  term  previously  developed  software  (PDS)  to  describe  software  that  falls  in 
one  of  three  categories; 

•  Commercial  Off-The-Shelf  (COTS)  software, 

•  Software  developed  to  a  standard  other  than  DO-178B,  and 

•  Software  developed  prior  to  DO-178B. 

By  this  definition,  it  is  hard  to  imagine  an  instance  when  a  product  service  history  argument  will 
be  made  on  software  other  than  PDS.  DO-178B  provides  specific  guidance  for  PDS  that  should 
be  used  in  conjunction  with  the  contents  of  this  handbook  when  seeking  certification  credit. 
Combining  alternative  methods  to  meeting  one  or  more  objectives  is  best  accomplished  by 
conducting  a  gap  analysis  designed  to  determine  where  data  may  be  insufficient  to  clearly 
demonstrate  compliance  with  the  objective.  Such  an  approach  is  described  in  DO-248B, 
discussion  paper  no.  5. 

2.4  PRODUCT  SERVICE  fflSTQRY  VERSUS  SQFTWAPF  T?P.T.IABILITY. 

The  DO-178B  definition  of  product  service  history  is  very  similar  to  the  IEEE  definition  of 
reliability,  which  is  “the  ability  of  a  product  to  perform  a  required  function  under  stated 
conditions  for  a  stated  period  of  time”.  It  should  also  be  noted  that  DO-178B  includes  the 
following  paragraphs  regarding  software  reliability: 

Section  2.2.3:  “Development  of  software  to  a  software  level  does  not  imply  the 
assignment  of  a  failure  rate  for  the  software.  Thus,  software  levels  or  software 
reliability  rates  based  on  software  levels  cannot  be  used  by  the  system  safety 
assessment  process  as  can  hardware  failure  rates.  ” 

Section  12.3.4;  “During  the  preparation  of  this  document  [DO-178B] ,  methods 
for  estimating  the  post-verification  probabilities  of  software  errors  were 
examined.  The  goal  was  to  develop  numerical  requirements  for  such  probabilities 
for  software  in  computer-based  airborne  systems  or  equipment.  The  conclusion 
reached,  however,  was  that  currently  available  methods  do  not  provide  results  in 
which  confidence  can  be  placed  to  the  level  required  for  this  purpose.  Hence,  this 
document  does  not  provide  guidance  for  software  error  rates.  If  the  applicant 
proposes  to  use  software  reliability  models  for  certification  credit,  rationale  for 
the  model  should  be  included  in  the  Plan  for  Software  Aspects  of  Certification, 
and  agreed  with  by  the  certification  authority.  ” 

The  effect  of  these  two  statements  has  been  a  virtual  moratorium  on  the  application  or  even 
exploration  of  software  reliability  as  an  alternative  method  for  satisfying  DO-178B. 

This  creates  an  inherent  difficulty  for  the  product  service  history  approach  as  well,  since  service 
history  arguments  are  largely  predicated  on  the  residual  error  rates  or  the  probability  of  latent 
software  errors  remaining  after  verification.  The  authors  of  DO-178B  side-stepped  this  issue  by 
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allowing  certification  credit  for  service  history  based  on  qualitative  assessments  of  the 
sufficiency  and  relevancy  of  the  product  service  history. 

^  THF.  FT  -FMENTS  OF  PRODUCT  SERVICE  fflSTORY. 

As  noted  in  the  previous  section,  the  topic  of  product  service  history  may  be  examined  by 
looking  at  the  various  elements  that  comprise  its  definition.  For  the  purposes  of  this  handbook, 
four  components  were  defined:  problem  reporting,  operations,  environment,  and  time. 
Considering  each  of  these  components  separately  results  in  different  but  interrelated  sets  of 
questions  that  must  be  asked  when  the  use  of  product  service  history  is  being  considered.  The 
questions  have  been  broken  into  these  classes  only  to  simplify  the  problem.  Answers  to  these 
questions  must  satisfy  both  the  relevancy  and  sufficiency  criteria  discussed  in  Section  12.3.5  of 
DO-178B. 

This  section  provides  a  discussion  of  each  set  of  questions  arising  fi-om  file  product  service 
history  definition.  One  representation  of  these  questions  is  provided  in  the  form  of  worksheets 
(see  £q)pendix  A).  While  these  worksheets  may  be  adapted  or  tailored  to  fit  a  particular  project, 
users  are  cautioned  to  maintain  an  objective  view  when  evaluating  service  history  data.  As 
illustrated  in  the  sections  below,  even  subtle  changes  in  any  one  of  the  four  areas  can  lead  to 
unpredictable  results  when  software  is  used  in  a  new  system  or  in  a  way  not  originally 
envisioned. 

3.1  QUESTIONS  OF  PROBLEM  REPORTING. 

Questions  of  problem  reporting  are  primarily  the  same  as  the  ones  faced  in  configuration  control 
and  management.  All  of  the  elements  of  DO-178B  Section  1 1.4  apply.  The  problems  have  to  be 
uniquely  identified,  they  should  be  traceable  to  the  version  of  software/product,  a  method  of 
closing  the  problems  must  be  defined,  and  closure  of  the  problems  must  be  accomplished  with 
proper  change  control  activity.  Changes  must  be  reviewed  for  priority  of  problems  and  change 
impact.  Data  on  problems,  corrections,  and  baselines  must  be  kept  under  control  to  assure  the 
integrity  of  the  data. 

All  of  these  activities  are  a  natural  part  of  a  well-defined  process.  However,  in  the  case  of 
previously  developed  software,  it  is  assumed  that  these  activities  are  not  visible  to  the  user  of  the 
product.  The  vendor  who  is  in  charge  of  problem  collection  may  not  have  a  robust  process.  The 
vendor  may  not  have  a  robust  policy  for  classifying  and  prioritizing  the  problems.  Multiple 
users,  employing  the  software  in  ways  to  which  die  vendor  has  no  visibility,  further  exacerbate 
this  issue. 

When  patches  are  installed,  some  users  may  install  the  patch,  whereas  many  others  may  not. 
This  means  that  some  users  are  using  the  imcorrected  version  and  some  are  using  the  corrected 
version.  This  results  in  service  history  that  cannot  be  treated  as  a  monolithic  block;  rather  it  must 
be  distributed  across  the  different  versions.  Only  those  versions  with  a  clear  similarity  to  the 
intended  use  may  be  used  to  arrive  at  the  total  product  service  history.  There  are  numerous 
reasons  affecting  problem  report  classification  and  accuracy  including: 

•  Not  all  users  may  be  using  the  software  product  per  the  user’s  manual. 


15 


•  Vendors  may  not  have  a  complete,  consistent,  or  accurate  way  of  identifying  those 
problems  attributable  to  software. 

•  Not  all  users  may  be  reporting  the  problems  they  encounter. 

•  Users  may  find  work-around  procedures  and  thus  stop  reporting  all  occurrences. 

•  Vendors  may  not  require  subsequent  occurrences  of  a  problem  to  be  reported. 

•  Vendors  may  treat  problems  found  internally  differently  than  those  found  by  their 
customers,  thus  xmderreporting  the  total  number  of  problems  experienced. 

Problems  may  also  be  introduced  while  fixing  other  problems.  Such  problems  should  also  be 
logged  once  the  product  is  fielded.  If  some  of  the  problems  are  unique  to  a  particular  small 
sector  of  users,  the  vendor  may  not  fix  the  problem  or  may  selectively  provide  a  patch.  Attention 
must  be  paid  to  the  number  and  type  of  open  problems.  A  vendor’s  policy  for  choosing  which 
errors  are  to  be  fixed  should  also  be  noted  in  the  qualitative  assessment.  A  vendor  may  place 
priority  on  non-safety-critical  problems  reported  by  a  large  sector  of  users  over  safety-critical 
problems  reported  by  a  small  sector  of  users. 

Assignment  of  version  numbers  and  tracking  the  operating  versions  of  the  product  to  be  traced  to 
the  problems  is  a  difficult  task.  If  vendors  provide  patches  for  their  software  or  fi'equently 
introduce  revisions  to  the  field,  this  must  be  taken  into  account  in  arriving  at  the  total  number  of 
versions  for  which  service  history  is  valid  and  for  which  the  total  service  periods  can  be 
combined. 

Visibility  into  how  problems  were  fixed  may  be  of  use  when  the  solutions  affect  the  usage  of  the 
product  in  a  safety-critical  application  (whether  requirements  were  compromised,  new 
assmnptions  were  made,  new  requirements  were  added,  new  design  features  were  added,  change 
impact  analysis  was  conducted,  list  of  affected  requirements/assumptions  are  provided  to  the 
user,  or  any  effect  on  hardware  is  noted,  etc.). 

Some  vendors  may  be  following  certain  regulations  or  policy  regarding  configuration  control  of 
the  problem  reporting  process.  Such  policies  may  help  in  determining  if  service  history  data  is 
clean.  Some  problems  may  also  be  corrected  in  periodic  upgrades  of  the  product.  It  is  important 
to  understand  the  vendor’s  policy  for  dissemination  of  patches,  warnings,  work-arounds,  and 
upgrades.  Spikes  in  error  rates  after  a  new  version  is  disseminated  need  to  be  traced  to  assess  the 
complexity  of  changes,  the  quality  of  change  impact  analysis,  the  quality  of  the  vendor’s 
verification  process,  and  the  diversity  of  the  product  usage. 

The  key  questions  to  be  addressed  in  the  area  of  problem  reporting  and  configuration 
management  for  the  purpose  of  establishing  the  minimum  objective  criteria  for  using  service 
history  data  include  a  good  consistent  definition  of  problems,  classification  of  problems,  tracking 
with  respect  to  software  versions,  and  tracking  with  respect  to  solutions. 
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Appendix  A,  table  A-2  provides  a  set  of  questions  in  the  form  of  a  worksheet  that  may  be  used  to 
evaluate  the  relevance  and  sufficiency  of  problem  report/configuration  management  using  the 
data  available  from  the  product  service  history. 

The  following  considerations,  based  on  Section  12.3.5  of  DO-178B,  were  used  in  formulating 
the  questions  in  appendix  A,  table  A-2: 

•  Data  available  on  problems 

•  Data  derivable  from  the  problem  reports 

•  Analysis  to  be  performed 

•  Indications  of  supplemental  verification 

3.2  QUESTIONS  OF  OPERATION. 

The  concept  of  operation  is  to  define  the  usage  characteristics  of  the  software  within  the  previous 
domain  as  compared  with  the  target  domain.  These  characteristics  include  people  and 
procedures  and  the  modes  in  which  the  service  history  was  documented  against  the  same  items 
within  the  target  domain.  Figure  1  illustrates  the  type  of  comparisons  that  are  needed. 
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FIGURE  1.  OPERATION 

There  are  many  concerns  in  evaluating  the  similarity  of  operations  that  may  not  be  so  obvious. 
Where  people  and  procedures  are  concerned,  the  training  and  qualifications  of  the  people  in  the 
service  history  domain  have  to  be  noted  so  that  the  proposed  domain  of  usage  can  account  for 
this  by  requiring  similar  training  and  qualification  requirements. 

Similarity  in  operational  modes  and  the  subset  of  software  fimctions  between  the  service  history 
domain  and  the  target  domain  are  the  main  focus  in  this  area.  In  general,  it  is  expected  that  the 
functions  to  be  employed  in  the  target  domain  are  a  subset  of  those  from  the  service  history 
domain.  Input  and  output  domains  may  be  evaluated  in  normal  and  abnormal  operations  to 
assess  the  completeness  of  coverage  of  all  of  the  functions  that  are  to  be  reused  in  the  target 
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domain.  This  is  the  most  fundamental  basis  for  getting  credit  for  service  history  by  assessing 
that  the  software  has  a  tried  and  tested  history  of  pertinent  functions. 

Consider  the  case  of  ARIANE  5.^  The  self-destruction  of  the  launcher  was  caused  by  tiie  failure 
to  mitigate  the  environmental  differences  between  ARIANE  5  and  ARIANE  4.  Software  reused 
in  ARIANE  5  fi’om  ARIANE  4  included  a  function  for  the  alignment  of  the  strap-down  inertial 
platform  to  be  operative  for  50  seconds.  This  requirement  was  based  on  a  particular  operational 
laimch  sequence  that  is  no  longer  used.  The  software  exception  generated  from  this  portion  of 
the  code  caused  a  chain  of  events  that  eventually  led  to;  the  backup  processor  shutting  off,  errors 
in  the  primary  processor  causing  an  angle  of  attack  of  more  than  20  degrees,  separation  of 
booster  on  the  main  stage,  and  self-destruction  of  the  launcher.  The  reused  function  should  have 
been  operative  only  before  liftoff  or  there  should  have  been  a  thorough  analysis  of  abnormal 
operating  modes,  differences  in  flight  operations,  and  nominal  range  and  value  of  parameters. 
There  should  have  been  a  discussion  of  software  exceptions  and  differences  in  actions  taken  to 
resolve  these  exceptions.  Questions  of  operation  and  environment  (discussed  next)  are  highly 
interrelated.  In  this  example,  a  study  of  target  operations  could  have  found  the  fault  just  as  easily 
as  a  study  of  the  target  environment. 

The  total  availability  of  service  history  data  may  be  much  longer  than  what  is  considered  similar 
operation.  For  example,  there  may  be  a  total  of  10,000  copies  of  a  particular  software  in  use  in 
the  public  domain,  out  of  which  only  10  copies  may  be  in  use  in  domains  similar  to  the  proposed 
usage.  This  would  have  a  direct  bearing  on  the  ability  to  calculate  error  rates.  This  is  discussed 
in  more  detail  in  the  Section  3.4,  Questions  of  Time,  of  this  handbook. 

Modifications  to  the  product  during  the  service  history  interval  need  to  be  studied  to  understand 
if  these  modifications  were  made  for  correcting  errors  in  a  dissimilar  domain  for  which  service 
history  credit  is  not  being  sought.  The  question  here  would  be  to  note  if  a  change  impact 
analysis  has  been  performed  to  assure  that  the  functions  that  are  of  consequence  in  the  service 
history  data  have  not  been  adversely  affected.  This  is  quite  possible  if  the  changes  have  affected 
either  the  assumptions  or  requirements  in  this  area. 

If  the  service  history  collection  occurred  when  the  software  was  being  used  at  a  lower  criticality 
than  the  intended  usage  in  the  target  domain,  caution  should  be  exercised  in  taking  credit.  The 
types  and  severity  of  errors,  as  well  as  open  problem  reports,  must  be  examined  to  assure  that  the 
service  history  gives  the  proper  level  of  assurance. 

It  must  be  noted  that  the  service  history  duration  should  ideally  include  both  normal  and 
abnormal  operations  to  cover  features  such  as  redundancy,  backup,  other  fault  tolerance 
techmques,  and  comer  conditions.  An  analysis  should  be  conducted  to  find  which  features  were 
not  exercised  in  the  service  history,  so  that  supplemental  verification  can  be  performed. 

If  the  product  was  used  with  different  data/parameters  (for  example  adaptation  data,  external  data 
sets,  internal  parameters)  in  the  service  environment,  these  differences  should  be  examined  for 
possible  risks  in  the  target  environment. 


^  “ARIANE  5  Flight  501  Failure,”  Reported  by  die  Inquiry  Board,  the  Chairman  of  the  Board,  Professor  J.  L.  Lions, 
Paris,  19  July  1996. 
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The  key  question  to  be  addressed  in  the  area  of  operation  for  the  purpose  of  establishing  the 
Tninimnm  objective  criteria  for  using  service  history  data  include  an  analysis  for  establishing 
similarity  of  operation  between  the  service  history  and  the  proposed  application.  Service  history 
data  that  reflect  dissimilar  operations  caimot  be  used  for  computing  service  history  duration. 

Appendix  A,  table  A-3  provides  a  set  of  questions  in  the  form  of  a  worksheet  that  may  be  used  to 
evaluate  the  similarity  of  operation  using  the  data  available  from  the  product  service  history. 

The  following  considerations,  based  on  Section  12.3.5  of  DO-178B,  were  used  in  formulating 
the  questions  in  appendix  A,  table  A-3  : 

•  Data  pertinent  to  operation 

•  Derivable  data  associated  with  operations 

•  Analysis  to  be  performed 

•  Indications  of  supplemental  verification 

3.3  QUESTIONS  OF  ENVIRONMENT. 

Questions  of  environment  were  broken  away  from  the  questions  of  operation  in  order  to 
distinguish  the  immediate  computer  environment  in  which  the  service  history  data  was  collected. 
This  particular  set  of  questions  are  designed  to  address  and  mitigate  software  errors,  which  have 
their  origin  in  hardware  errors,  interface  errors,  or  resource  assumptions.  It  should  be  noted  that 
the  exception  handling  and  fault  tolerance  of  the  product,  whose  service  history  is  being  tracked, 
should  be  separated  from  the  larger  system  in  which  it  is  embedded  so  that  assurance  is  gained 
on  the  robustness  of  the  product.  This  knowledge  allows  for  an  appropriate  reuse  of  the  product 
in  the  new  environment.  Figure  2  illustrates  the  items  that  should  be  considered  in  this  area. 
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FIGURE  2.  ENVIRONMENT 
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Similarity  of  environment  may  be  assessed  using  the  history  of  modifications  to  the  product  due 
to  the  particular  hardware  platform  or  because  of  resource  requirements  in  the  service 
environment  or  the  similar  types  of  modifications  made  to  the  product  in  the  target  environment. 

Consider  the  example  of  the  Patriot  systems  “failure”  to  intercept  the  El  Hussein  (Scud)  missile 
in  Deharan.  Operational  specifications  for  the  system  matched  with  the  way  the  system  behaved. 
However,  the  “problems”  in  the  software  were  bugs  only  AFTER  the  operational  environment 
had  been  redefined.  The  weapon  was  used,  not  in  the  detection  and  interception  of  aircraft,  but 
rather  in  the  detection  and  interception  of  land-launched  missiles.  In  its  new  capacity,  the 
software  “failed”  because  (1)  there  were  missiles  in  a  speed  range  that  could  and  should  be 
attacked  and  (2)  the  Patriot  system’s  primary  mission  would  NOT  be  defending  against  hostile 
aircraft  over  a  relatively  short  attack  time,  but  rather,  defending  against  a  potential  land-launched 
missile  threat  over  many  days.  System  performance  degradation  due  to  uncompensated  clock 
drift  crippled  the  weapon’s  defensive  capability  after  the  system  had  been  continuously  powered 
for  days  rather  than  the  hours  it  was  designed  for^.  Unlike  the  ARIANE  case,  it  would  have  been 
difficult  to  detect  the  “problems”  in  this  case  since  the  system’s  failure  was  ultimately  tied  to  the 
overall  environment  definition. 

Service  history  credit  should  be  counted  strictly  when  the  types  of  installations  match  the  target 
environment;  i.e.,  same  or  similar  hardware  platforms.  Product  literature  may  be  reviewed  to 
compare  computer  environments  in  terms  of  limitations  and  constraints  such  as  resource  usage. 

If  the  problem  reports  identify  problems  because  of  usage  in  a  particular  computer  environment 
differ  firom  the  target  environment  and  the  changes  were  made  to  fix  these  problems,  the  effect  of 
ttiese  changes  in  the  target  environment  should  be  considered. 

If  the  product  was  used  with  different  data/parameters  (for  example  adaptation  data,  external  data 
sets,  internal  parameters)  in  the  service  environment,  these  differences  should  be  examined  for 
possible  risks  in  the  target  environment. 

Uie  key  questions  to  be  addressed  in  the  area  of  environment  include  assessing  the  computing 
environment  to  assure  that  the  environment  in  which  the  software  was  hosted  during  service 
history  is  similar  to  the  proposed  environment.  This  analysis  must  include  not  just  object  code 
compatibility,  but  time  and  memory  utilization,  accuracy,  precision,  communication  services, 
built-in  tests,  fault  tolerance,  channels,  ports,  queuing  models,  priorities,  error  recovery  actions, 
etc. 

Appendix  A,  table  A-4  provides  a  set  of  questions  in  the  form  of  a  worksheet  that  may  be  used  to 
evaluate  the  similarity  of  environment  using  the  data  available  from  the  product  service  history. 


^  Patriot  Missile  Defense  Software  Problems  led  to  Systems  failure  at  Dhahran,  Saudi  Arabia,  GAO  Report 
February,  1992,  B-247094. 
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The  following  considerations,  based  on  Section  12.3.5  of  DO-178B,  were  used  in  formulating 
the  questions  in  appendix  A,  table  A-4: 

•  Data  pertinent  to  the  computer  environment 

•  Derivable  data  associated  with  the  computer  environment 

•  Analysis  to  be  performed 

•  Indications  of  supplemental  verification 

3.4  QUESTIONS  OF  TIME. 

There  are  many  different  ways  of  measuring  the  service  history  duration;  duration  may  be 
measured  without  considering  the  changes  made  to  the  software  or  the  clock  may  be  restarted  at 
the  time  corrections  are  made  to  safety  problems.  This  question  is  related  to  how  problems  are 
classified  and  the  vendor’s  priority  system  for  correcting  the  problems.  Figure  3  illustrates  one 
common  approach  to  measuring  service  history  duration. 

Maximum  Product  Service  History 
Available  (single  instance) 

— z - Z — 

PSAC-  SAS- Error 

Error  Rates 

Rates  Validated  or 

Defined  Renegotiated 

FIGURE  3.  TIMELINE 

The  question  of  defining  time  relative  to  certification  or  continuing  airworthiness  has  its  parallels 
in  other  areas  of  the  FAA.  For  example,  following  the  Aloha  Airlines  incident  in  1988,  the 
National  Transportation  Safety  Board  noted,  as  part  of  their  findings,  that  there  appeared  to  be 
confusion  in  the  terms  flight  cycle  versus  flight  hour.  The  FAA  released  a  Flight  Standards 
Handbook  Bulletin  (HBAW  94-05B)  to  address  this  confusion  as  it  related  to  aircraft 
maintenance. 

The  premise  for  using  service  history  is  based  on  the  assumption  that  service  history  data  gives 
evidence  that  all  of  the  required  functions  have  been  repeatedly  exercised  and  is  correct.  Strictly 
speaking,  this  assumption  has  no  bearing  on  time  at  all.  Time  comes  into  the  picture  only 
because  there  is  comfort  in  a  statistical  sense  that  the  probability  of  exercising  all  of  tibe  needed 
functions  is  greater  as  more  time  passes. 

Time  is  further  modified  within  the  definition  by  the  need  for  its  measurement  to  take  place  over 
a  ‘contiguous’  period.  This  qualification  is  designed  to  eliminate  the  potential  for  periods  of 
improper  execution  or  dormancy  to  be  suppressed,  thus  skewing  any  conclusions  drawn  about 
the  software  under  consideration. 
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A  close  review  of  the  DO-178B  product  service  history  guidance  produces  additional  terms  that 
directly  relate  to  time,  most  notably  “hours  in-service”  and  “normal  operation  time.”  These 
sound  suspiciously  like  terms  used  to  arrive  at  reliability  numbers  for  hardware.  An  applicant 
wishing  to  use  product  service  history  is  asked  to  describe  in  their  Plan  for  Software  Aspects  of 
Certification  (PS  AC)  the  rationale  for  choosing  a  particular  number  of  hours  in-service  including 
how  normal  operation  time  is  defined.  Consider  a  software  function  that  is  only  used  during 
landing.  It  hardly  seems  reasonable  to  define  in-service  hours  as  flight  time  when  the  landing 
phase  during  which  the  software  is  being  exercised  accounts  for  only  a  small  fraction  of  this 
overall  time. 

While  DO-178B  is  silent  on  whether  the  contiguous  time  period  varies  with  software  level,  all  of 
the  discussions  within  SC-190,  SC-180,  and  CAST  have  tended  to  accept  this  as  an  axiom. 
Likewise,  the  assumption  is  always  made  that  what  is  being  discussed,  in  some  way,  is 
measurable  using  hours,  minutes,  days,  etc.  It  is  generally  felt  that  attempting  to  categorize 
software  execution  in  terms  of  clock  cycles,  frames,  or  states  is  generally  something  for  which 
sufficient  data  would  be  impossible  to  directly  measure  and  would  ultimately  rely  on  inference 
from  a  clock-based  measurement. 

DO-178B  in  Section  12.3.5  j.  (2)  and  k  refer  to  computation  of  error  rates.  DO-178B  does  not 
provide  specific  guidance  as  to  how  this  computation  should  be  performed  or  what  specific  data 
is  to  be  used.  This  provides  the  applicant  with  a  fair  amount  of  flexibility  in  the  application  of 
the  service  history  argument.  Error  rates  could  be  defined  as  number  of  errors  divided  by  the 
time  duration.  In  some  cases,  time  duration  is  not  as  relevant  as  using  number  of  events  such  as 
takeoffs  or  landing,  flight  hours,  fli^t  distance,  total  population  operating  time,  or  only  the 
number  of  times  an  operator  queried  the  software  for  specific  information.  For  use  in  this 
computation,  the  duration  should  be  analyzed  to  be  relevant.  DO-178B  does  not  specify  whether 
all  errors  are  considered  to  be  of  the  same  weight  in  these  computations.  It  seems  logical  that 
even  a  few  safety  errors  should  be  of  higher  consequence  than  a  large  number  of  nonsafety 
errors.  Although  there  is  a  discussion  of  safety  problems  in  Section  12.3.5  h,  there  is  no 
indication  of  how  these  problems  are  used  in  error  rate  computations. 

Note:  Grounds  for  restarting  the  clock  within  the  service  history  duration  is  not  discussed  in 
DO-178B.  When  a  major  software  or  hardware  change  occurs  a  key  question  must  be 
answered.  The  key  question  to  answer  is  whether  service  history  duration  should  be 
measured  before  or  after  the  implementation  of  the  changes.  The  measurement  of  the 
error  rates  for  the  updated  software  or  hardware  is  dependent  upon  the  answer  to  this 
question.  In  a  number  of  software  reliability  models,  time  is  reset  when  major  changes 
are  made  since  the  software  that  was  tracked  is  no  longer  the  software  that  is  used.  There 
are  other  models  that  compensate  for  changes  in  software.  This  gap  is  tied  to  whether 
software  reliability  models  can  be  used,  and  if  so,  how  do  you  assess  that  the  correct 
assumptions  are  made  in  the  use  of  a  particular  model  in  a  particular  circumstance.  This 
is  illustrated  in  figure  4. 
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FIGURE  4.  CALCULATION  OF  SERVICE  HISTORY  DURATION 

The  key  questions  to  be  addressed  in  the  area  of  time  for  the  purpose  of  establishing  the 
minimum  objective  criteria  for  using  service  history  data  include  units  of  measurement  in  the 
service  history  duration  definition  as  jq)propriate  to  the  usage  of  the  software  in  question, 
reliability  and  consistency  of  measurement  of  this  time,  and  justification  for  duration  used  in  the 
calculation  of  error  rates. 

Appendix  A,  table  A-5  provides  a  set  of  questions  in  the  form  of  a  worksheet  that  may  be  used  to 
evaluate  service  history  time  duration  and  error  rates  using  the  data  available  in  the  product 
service  history. 

The  following  considerations,  based  on  Section  12.3.5  of  DO-178B,  were  used  in  formulating 
the  questions  in  appendix  A,  table  A-5: 

•  Pertinent  data  related  to  time 

•  Derivable  data  regarding  time 

•  Error  rate  considerations 

•  Analysis  to  be  performed 

•  Indications  of  supplemental  verification 

4.  ADEQUACY  OF  DEVELOPMENT  PROCESS. 

DO-178B  gives  guidance  for  both  process  and  product  assurance.  Product  service  history  does 
not  provide  any  direct  objective  evidence  of  the  process  used  in  creating  the  software. 
Applicants  wisWg  to  make  use  of  product  service  history  must  determine  a  way  of 
demonstrating  compliance  with  the  objectives  of  DO-178B.  This  generally  involves 
complementing  product  service  history  with  additional  alternate  methods. 

Nmnerous  attempts  have  been  made  to  equate  specific  objectives  for  which  product  service 
history  could  be  “traded  with.”  Such  attempts  within  SC-190  and  CAST  actually  involved  the 
creation  of  tables  listing  the  objectives  of  DO-178B  and  stating  for  each  objective  whether 
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service  history  data  that  could  fiilly  or  partially  satisfy  the  objective.  These  attempts  were 
reviewed  as  part  of  this  research  in  hopes  that  something  had  been  overlooked  that  prevented 
their  acceptance  by  the  broader  community.  The  inherit  problem  is  the  unquantifiable  nature  of 
die  processes  used  to  create  and  assure  software.  DO-178B  is  based  on  the  premise  that  a  good 
development  process  yields  a  better  product;  one  that  is  more  likely  to  perform  its  intended 
function  and  less  likely  to  perform  unintended  functions. 

The  66  objectives  of  DO-178B  are  divided  across  six  main  areas:  planning,  development, 
verification,  quality  assurance,  configuration  management,  and  certification  liaison.  The 
definition  of  product  service  history  really  only  addresses  two  of  these.  The  first  is  fairly  direct, 
namely,  problem  reporting  and  configuration  management  of  the  software  the  data  describes. 
The  second  is  verification  of  the  code  to  some  degree  by  virtue  of  its  execution.  In  fact,  a  cogent 
argument  can  be  made  that  service  history  represents  a  variety  of  testing  techniques,  including: 

•  Stress  testing 

•  Random  testing 

•  Scenario-based  testing 

•  Regression  testing 

•  Accelerated  life  testing 

•  Exhaustive  testing 

•  Domain  testing 

•  Error  guessing 

All  of  these  techmques  may  be  used  to  accomplish  one  or  more  of  the  verification  objectives 
outlined  in  DO-178B.  These  techniques  fi-equently  are  applied  to  the  elements  of  blackbox 
testing  in  controlled  environments,  either  laboratory  or  airplane,  in  typical  DO-178B  projects. 
The  good  news  is  that  about  60%  of  the  objectives  in  DO-178B  are  verification  objectives.  The 
bad  news  is  that  there  would  not  seem  to  be  any  corollary  to  product  service  history  for  planning, 
development,  quality  assurance,  and  certification  liaison  during  the  original  development  of  the 
software  that  the  service  history  data  describes. 

With  this  in  mind,  it  would  seem  most  appropriate  to  focus  specific  attention  on  things  that  may 
be  done  to  gain  confidence  in  these  other  areas.  If  any  development  records  are  still  available 
firom  the  original  vendor,  these  may  be  reviewed  to  gain  confidence  in  the  process  that  was 
followed.  Such  records  could  include  the  requirements  documents,  design  data,  quality 
assurance  data,  and  test  data  that  may  supplement  the  service  history  data.  In  this  last  case, 
special  attention  should  be  paid  to  testing  completed  for  error-handling  routines,  performance 
testing,  and  other  testing  focused  on  the  robustness  characteristics  of  the  software.  Remember 
that  these  are  the  parts  of  the  code  least  likely  to  have  been  exercised  as  part  of  the  service 
history. 

Confidence  in  the  supplier’s  development  process  may  also  be  gained  through  careful  analysis  of 
the  problem  report  data  collected  over  &e  service  history  period.  In  addition  to  the  items 
discussed  at  the  beginning  of  section  3,  trend  data  may  be  analyzed  to  determine  how  well  the 
supplier  accomplishes  reverification  and  whether  the  software  does,  in  fact,  appear  to  be 
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maturing  over  time.  This  type  of  analysis  is  not  directly  discussed  in  the  DO-178B  but  is 
generally  accepted  by  the  software  community. 

Note  that  each  of  the  above  approaches  can  be  stated  in  the  negative  as  well.  Spikes  in  problem 
r^orts  right  after  a  major  build  or  a  patch  may  indicate  that  the  software  is  not  maintainable  or 
the  quality  of  updates  is  not  quite  high  enough.  Recmring  or  chronic  problems  that  go 
unresolved  may  also  indicate  poor  processes. 

5.  ESTABLISHMENT  OF  “EQUIVALENT  SAFETY”. 

Within  a  month  and  a  half  of  its  publication,  DO-178B  was  formally  recognized  via  AC  20-1 15B 
as  a  means,  but  not  the  sole  means,  for  securing  FAA  approval  of  software  in  airborne  systems. 
For  new  projects  started  after  this  AC  was  published,  most  applicants  have  chosen  to  use  DO- 
178B  as  the  means  of  compliance  for  their  airborne  software.  Those  who  have  sought  to  use 
other  approaches  for  securing  FAA  approval  have  generally  been  required  to  show  how  their 
approach  met  the  intent  behind  the  DO-178B  objectives. 

One  of  the  most  basic  issues  when  discussing  product  service  history  is  to  understand  what  that 
service  history  is  demonstrating.  Since  the  service  history  data  generally  exists  for  a  system, 
typically  a  line-replaceable  unit  on  an  aircraft,  any  claim  made  for  the  software  is  an 
extrapolation  from  the  system’s  performance.  Systems  are  required  to  comply  with  one  or  more 
CFR  before  being  certificated  for  use  on  an  aircraft.  A  careful  reading  of  DO-178B  along  with 
the  guidance  governing  certification  of  parts  and  equipment  described  in  14  CFR  Part  21  shows 
that  DO-178B  is  simply  a  means  of  satisfying  the  CFRs,  specifically  those  elements  describing 
intended  function  and  absence  of  unintended  function  as  noted  earlier.  The  logical  question  that 
arises  is  whether  service  history  can  be  used  to  demonstrate  compliance  directly  witii  the  CFRs. 

While  current  guidance  does  not  preclude  such  an  argument,  actually  being  able  to  demonstrate 
CFR  compliance  would  be  extremely  difficult.  Any  attempt  would  need  to  overcome  the  basic 
issue  of  reliability  applied  to  software.  CFR  XX.  1309  uses  terms  such  as  extremely  improbable 
to  describe  events  that  simply  should  not  happen  in  the  lifetime  of  a  particular  aircraft  type.  This 
has  historically  been  translated  to  failure  probabilities  of  10'^  or  better.  There  exists  no 
commercially  accepted  model  for  software  reliability  that  comes  close  to  this  number  and  that 
can  be  shown  to  be  based  on  a  realistic  model. 

A  component  of  unknown  pedigree  within  a  system  is  a  safety  risk.  Contribution  to  safety  from 
the  so:^are  components  of  the  system  has  been  imder  constant  debate  since  software  was  first 
introduced  in  a  Flight  Management  System  in  the  early  1980s.  For  the  time  being,  design 
assurance  remains  the  only  viable  approach,  with  DO-178B  serving  as  the  most  mature  model 
for  its  application.  There  are,  however,  other  ways  of  mitigating  risk  from  an  unknown  software 
component.  For  example,  architectural  means  may  be  employed  to  limit  the  effect  of  a  software 
error  leading  to  a  system-level  failure.  Examples  of  architectural  means  include; 

•  Partitioning — ^preventing  failmes  from  noncritical  functions  affecting  critical  functions 
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“Wrappers” — ^wrapper  software  prevents  use  of  unneeded  functionality,  checks  validity 
of  parameters 


•  Software  Performance  and  Safety  monitors — credibility  checks;  redundant  processors 
checking  one  another,  fail-safe  architectures 

•  External  monitoK — e.g.,  watchdog  timers 

Unfortunately,  architectural  means  may  not  always  be  an  option  to  protect  against  latent  errors  in 
software  for  which  only  service  history  data  is  available  for  certification  credit.  It  may  also  be 
that  the  use  of  architectural  means  actually  increases  the  system’s  complexity,  thus  potentially 
decreasing  safety  rather  than  increasing  it.  For  higher  criticality  systems,  service  history  may 
simply  not  be  an  appropriate  or  practical  choice. 

It  is  generally  accepted  that  use  of  service  history  data  becomes  easier  when  systems  are 
relatively  simple  and  of  lower  criticality,  the  service  history  data  is  both  abundant  and  of  high 
quality,  and  the  system’s  operating  characteristics  and  external  environment  are  close  to  the 
original  systems. 

6.  SUMMARY. 

Service  history  is  a  powerful  approach,  which  when  used  correctly,  can  make  it  possible  to 
demonstrate  the  maturity  of  software  that  has  previously  been  fielded  and  for  which  good  data 
regarding  its  operation  is  available.  To  accomplish  this,  careful  attention  must  be  paid  to  a 
number  of  questions  concerning  the  application  of  service  history.  Section  3,  The  Elements  of 
Product  Service  History,  of  this  handbook,  discussed  these  questions  in  detail.  Sections  4, 
Adequacy  of  Development  Process,  and  Section  5,  Establishment  of  “Equivalent  Safety,”  helped 
to  place  those  questions  in  the  context  of  design  assurance  and  safety;  two  ftmdamental  aspects 
of  creating  and  assuring  software  for  airborne  systems  and  equipment.  In  appendix  A,  a  detailed 
set  of  worksheets  are  provided  to  aid  applicants  in  answering  the  questions  relating  to  service 
history  and  to  provide  a  firamework  for  the  necessary  dialogue  with  the  certification  authority. 

Service  history,  as  an  alternate  method  for  demonstrating  compliance  to  the  objectives  of 
DO-178B,  is  only  one  of  many  approaches  that  may  be  taken  to  demonstrate  software  maturity. 
When  contemplating  its  use,  one  must  be  careful  to  consider  the  relationship  between  service 
history  and  software  reliability.  As  noted  in  section  2  software  reliability  remains  a  controversial 
idea  and  cannot  be  used  quantitatively  in  a  safety  assessment.  Careful  attention  must  be  applied 
when  defining  error  rates  for  a  particular  application  and  their  definition  should  be  discussed 
with  the  appropriate  Aircraft  Certification  Office  (ACO)  at  the  onset  of  the  certification  project. 

As  more  confidence  is  gained  in  the  use  of  service  history  arguments  for  supporting  certification 
efforts,  the  FAA  may  develop  additional  guidance.  It  is  also  expected  that  this  subject  will  be 
revisited  when  DO-178B  is  revised  in  the  future.  In  the  interim,  it  is  hoped  that  this  report  will 
help  applicants  apply  service  history  arguments  in  a  more  thorou^  and  consistent  manner 
Likewise,  the  use  of  this  handbook  by  the  FAA  should  allow  for  more  consistent  expectations 
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being  placed  on  applicants,  something  that  has  generally  been  shown  to  help  control  costs 
associated  with  achieving  certification. 
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APPENDIX  A— EVALUATION  WORKSHEETS 


The  following  worksheets  are  designed  to  provide  a  uniform  and  consistent  mechanism  for 
conducting  an  evaluation  of  product  service  history  using  the  questions  discussed  in  sections  3 
through  5  of  this  handbook.  Questions  may  need  to  be  added  or  tailored,  depending  on  a 
particular  project  or  through  discussions  with  the  appropriate  Aircraft  Certification  Office. 

These  worksheets  contain  the  questions  derived  from  Section  12.3.5  of  DO-178B  and  as 
discussed  in  tables  A-1  toough  A-4  of  this  handbook.  Those  questions  without  a  DO-178B 
reference  originated  from  other  industry  sectors  (OIS)  that  make  use  of  service  history  for  the 
purposes  of  evaluation  and  approval  and  are  indicated  by  OIS.  Since  these  represent  the  best 
practices  for  the  application  of  service  history  arguments,  they  have  been  included  here  for 
completeness. 


TABLE  A-1.  WORKSHEET  FOR  QUESTIONS  OF  PROBLEM  REPORTING 


Area: 

Problem  Reporting/ 
Configuration  Management 

Software  Being 
Evaluated: 

Project: 

Evaluator: 

Date: 

DO-178B 

Reference 

Question 

Response 

Issues 

1. 

12.3.5  a 
and  c 

Are  the  software  versions  tracked  during 
the  service  history  duration? 

2. 

12.3.5  a 
andc 

Are  problem  reports  tracked  with  respect 
to  particular  versions  of  software? 

3. 

12.3.5  a 

Are  problem  reports  associated  with  the 
solutions/patches  and  an  analysis  of 
change  impact? 

4. 

12.3.5  a 

Is  revision/change  history  maintained  for 
different  versions  of  the  software? 

5. 

12.3.5  a 

Have  change  impact  analyses  been 
performed  for  changes? 

6. 

12.3.5  b 

Were  in-service  problems  reported? 

7. 

12.3.5  b 

Were  all  reported  problems  recorded? 

8. 

12.3.5  b 

Were  these  problem  reports  stored  in  a 
rq)Ository  from  which  they  can  be 
retrieved? 

9. 

12.3.5  b 

Were  in-service  problems  thoroughly 
analyzed,  and/or  those  analyses  included 
or  appropriately  referenced  in  the  problem 
reports? 

10. 

OIS 

Are  problems  within  problem  report 
repository  classified? 
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TABLE  A-1.  WORKSHEET  FOR  QUESTIONS  OF  PROBLEM  REPORTING  (Continued) 


DO-178B 

Reference 

Question 

Response 

Issues 

11. 

OIS 

If  the  same  type  of  problem  was  reported 
for  multiple  times,  were  there  multiple 
entries  or  a  single  entry  for  a  specific 
problem? 

12. 

OIS 

If  problems  were  found  in  the  lab  in 
executing  copies  of  operational  versions  of 
software  during  the  service  history  period, 
were  these  problems  included  in  the 
problem  reporting  system? 

13. 

12.3.5  c 

Is  each  problem  report  tracked  with  the 
status  of  whether  it  is  fixed  or  open? 

14. 

12.3.5  c 

If  the  problem  was  fixed,  is  there  a  record 
of  how  the  problem  was  fixed  (in 
requirements,  design,  code)  ? 

15. 

12.3.5  c 

Is  there  a  record  of  the  new  version  of 
software  with  a  new  release  after  the 
problem  was  fixed? 

16. 

12.3.5  c 

Are  there  problems  with  no  corresponding 
record  of  change  in  software  version? 

17. 

12.3.5  c 

Does  the  change  history  show  that  the 
software  is  currently  stable  and  mature? 

18. 

OIS 

Does  the  product  have  the  property  of 
exhibiting  the  error  with  message  to  the 
user?  (Some  products  may  not  have  error¬ 
trapping  facilities,  so  they  may  just 
continue  executing  with  wrong  results 
with  no  indication  of  failure.) 

■ 

OIS 

Has  the  vendor  (or  the  problem  report 
collecting  agency)  made  it  clear  to  all 
users  that  problems  are  being  collected  and 
corrected? 

20. 

12.3.5  h 

Are  all  problems  within  a  problem  report 
repository  classified? 

21. 

12.3.5  h 

Are  safety-related  problems  identified  as 
such?  Can  safety-related  problems  be 
retrieved? 

22. 

12.3.5  h 

Is  there  a  record  of  which  safety  problems 
are  fixed  and  which  problems  remain 
open? 
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TABLE  A-1 .  WORKSHEET  FOR  QUESTIONS  OF  PROBLEM  REPORTING  (Continued) 


DO-178B 

Reference 


12.3.5  h 


24. 

12.3.5  h 

25. 

OIS 

26. 

12.3.5  i 

27. 

12.3.5  i 

28. 

12.3.5  i 

29. 

OIS 

30. 

OIS 

31. 

OIS 

32. 

12.3.5  j(2) 

33. 

12.3.5  j(2) 

34. 

OIS 

35. 

12.3.5  j(2) 

Question _ 


Is  there  enough  data  after  the  last  fix  of 
safety-related  problems  to  assess  that  the 
problem  has  been  corrected  and  that  no 
new  safety-related  problems  have 
surfaced?  _ 


Do  open  problem  reports  have  any  safety 
impact? _ 


Is  there  enough  data  after  the  last  fix  of 
safety-related  problems  to  assess  that  the 
problem  is  solved  and  that  no  new  safety- 
related  problems  have  surfaced? _ 


Are  the  problem  reports  and  their  solutions 
classified  to  indicate  how  a  fix  was 
implemented? _ 


Is  it  possible  to  trace  particular  patches  to 
specific  release  versions  and  infer  firom 
design  and  code  fixes  that  the  new 
versions  correspond  to  these  fixes? _ 


Is  it  possible  to  separate  the  problem 
reports  that  were  fixed  in  the  hardware  or 
change  of  requirements? _ 


Are  problem  reports  associated  with  the 
solutions/patches  and  an  analysis  of 
change? 


If  the  solutions  indicated  a  change  in  the 
hardware  or  mode  of  usage  or 
requirements,  is  there  an  analysis  of 
whether  these  changes  invalidate  the 
service  history  data  before  that  change? 


Is  there  a  fix  to  a  problem  with  changes  to 
software  but  with  no  record  of  change  in 
the  software  version? 


Is  the  service  period  defined  appropriate  to 
the  nature  of  Ae  software  in  question? 


How  many  copies  of  the  software  are  in 
use  and  being  tracked  for  problems? _ 


How  many  of  these  applications  can  be 
considered  to  be  similar  in  operation  and 
environment? 


Are  the  input/oulput  domains  the  same 
between  the  service  duration  and  the 
proposed  usage? _ 


Response 


Issues 


TABLE  A-1.  WORKSHEET  FOR  QUESTIONS  OF  PROBLEM  REPORTING  (Continued) 


DO-178B 

Reference 

Question 

Response 

Issues 

36. 

OIS 

If  the  input/output  domains  are  different, 
can  they  be  amended  using  glue  code? 

37. 

12.3.5  j(2) 

Does  the  service  period  include  normal 
and  abnormal  operating  conditions? 

38. 

OIS 

Is  there  a  record  of  the  total  number  of 
service  calls  received  during  the  period? 

39. 

OIS 

Were  warnings  and  service  interruptions  a 
part  of  this  problem-reporting  system? 

40. 

OIS 

Were  warnings  analyzed  to  assure  that 
they  were  or  were  not  problems? 

41. 

12.3.5  j-(3) 

Was  there  a  procedure  used  to  log  the 
problem  reports  as  errors? 

42. 

12.3.5  j(3) 

What  was  the  reasoning  behind  the 
contents  of  the  procedure? 

43. 

OIS 

Is  there  evidence  that  this  procedure  was 
enforced  and  used  consistently  throughout 
the  service  history  period? 

44. 

OIS 

Does  the  history  of  warranty  claims  made 
on  the  product  match  with  die  kind  of 
problems  seen  in  the  service  history? 

45. 

OIS 

Have  problem  reports  identified  as  a 
nonsafety  problem  in  the  original  domain 
been  reviewed  to  determine  if  they  are 
safety-related  in  the  target  domaiu? 
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TABLE  A-2.  WORKSHEET  FOR  QUESTIONS  OF  OPERATION 


Area:  Operation 


Project: 


Date: 


Software  Being 
Evaluated: 


Evaluator: 


12.3.5  d 


12.3.5  e 


12.3.5  g 


12.3.5  j(l),g 


12.3.5  j(2) 


Question _ 


Is  the  intended  software  operation 
similar  to  the  usage  during  the 
service  history  (its  interface  with  the 
external  “world,”  people,  and 
procedures)? _ 


Have  the  differences  between  service 
history  usage  and  proposed  usage 

been  analyzed? _ 

Are  there  differences  in  the  operating 

modes  in  the  new  usage? _ 

Are  only  some  of  the  fimctions  of  the 
proposed  application  used  in  service 
usage? _ _ 


Is  there  a  gap  analysis  of  functions 
that  are  needed  in  the  proposed 
application  but  have  not  been  used  in 
the  service  duration? 


Is  the  definition  of  “normal 
operation”  and  “normal  operation 
time”  appropriate  to  the  product? 


Does  service  period  include  normal 
and  abnormal  operating  conditions? 

Is  there  a  technology  difference  in  the 
usage  of  product  firom  service  history 
duration  (manual  vs  automatic,  user 
intercept  of  errors,  used  within  a 
network  vs  standalone,  etc.)?  _ 


Was  operator  training  on  procedures 
required  in  the  use  of  product  during 
the  recorded  service  history  time 
period? _ 


Is  there  a  plan  to  provide  the  similar 
training  in  the  new  operation? _ 


Will  the  software  level  for  the  new 
system  be  the  same  as  it  was  in  the 
old  system? 


Response 


Issues 
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TABLE  A-3.  WORKSHEET  FOR  QUESTIONS  OF  ENVIRONMENT 


Area:  Environment 


Project: 


Software  Being 
Evaluated: 


Evaluator: 


DO-178B 

Reference 

Question 

12.3.5  e 

Is  the  hardware  environment  of  service  history 
and  the  target  environment  similar? 

OIS 

Have  the  resource  differences  between  the  two 
computers  been  analyzed  (time,  memory, 
accuracy,  precision,  communication  services, 
built-in  tests,  fault  tolerance,  channels  and 
ports,  queuing  modes,  priorities,  error  recovery 
actions,  etc.)? 

OIS 

Are  safety  requirements  encountered  by  the 
product  the  same  in  both  environments? 

OIS 

Are  exceptions  encountered  by  the  product  the 
same  in  both  enviromnents? 

Response 


Issues 


Is  the  data  needed  to  analyze  the  similarity  of 
the  environments  available?  (Such  data  are  not 
usually  a  part  of  problem  data.) 


OIS 

Does  the  analysis  show  which  portions  of  the 
service  history  data  are  applicable  to  the 
proposed  use? 

OIS 

How  much  service  history  credit  can  be 
assigned  to  the  product,  as  opposed  to  the  fault 
tolerant  properties  of  the  computer 
environment  in  the  service  history  duration? 

OIS 

Is  the  product  compatible  with  the  target 
computer  without  making  modifications  to  the 
product  software? 

12.3.5  e 
andj(2) 

If  the  hardware  environments  are  different, 
have  file  differences  been  analyzed? 

OIS 

Were  there  hardware  modifications  during  the 
service  history  period? 

OIS 

If  there  were,  is  it  still  appropriate  to  consider 
the  service  history  duration  before  the 
modifications? 

OIS 

Are  software  requirements  and  design  data 
needed  to  analyze  whether  the  configuration 
control  of  any  hardware  changes  noted  in  the 
service  history  are  acceptable? 
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TABLE  A-4.  WORKSHEET  FOR  QUESTIONS  OF  TIME 


Area:  Time 


Project: 


Date: 


12.3.5  j(4) 


15.  12.3.5  j(4) 


6.  lOIS 


Software  Being 
Evaluated: 


Evaluator: 


DO-178B 

Reference 

Question 

IkftitfilEH 

What  is  the  definition  of  service  period? 

12.3.5  j(2) 

Is  the  service  period  defined  appropriate  to  the 
nature  of  software  in  question? 

12.3.5  j(2) 

What  is  the  definition  of  normal  operation 
time? 

12.3.5  j(2) 

Does  normal  operation  time  used  in  the  service 
period  include  normal  and  abnormal  operating 
conditions? 

Glossary 

Can  contiguous  operation  time  be  derived  from 
service  history  data? 

OIS 

Is  the  “applicable  service”  portion  recognized 
from  the  total  service  history  data  availability? 

12.3.5  j(2) 

What  was  the  criterion  for  evaluating  service 
period  duration? 

12.3.5  j(2) 

How  many  copies  of  the  software  are  in  use 
and  being  tracked  for  problems? 

OIS 

What  is  the  duration  of  applicable  service? 

11. 

OIS 

12. 

OIS 

13. 

OIS 

Is  the  applicable  service  definition  appropriate? 


Is  this  the  duration  used  for  calculation  of  error 
rates?  _ 


How  reliable  was  the  means  of  measuring 
time?  _ 


How  consistent  was  the  means  of  measuring 
time  throughout  the  service  history  duration? 


Do  you  have  a  proposed  accepted  error  rate  that 
is  justifiable  and  appropriate  for  the  level  of 
safety  of  proposed  usage,  (before  analyzing  the 
service  history  data)? _ 


How  do  you  propose  that  this  error  rate  be 
calculated?  (Before  analyzing  the  service 
history  data! _ 


Is  the  error  rate  computation  (total  errors 
divided  by  time  duration,  number  of  execution 
cycles,  number  of  events  such  as  landing,  flight 
hours,  flight  distance,  or  by  total  population 
operating  time)  appropriate  to  the  application 
in  question? _  _ 


TABLE  A-4.  WORKSHEET  FOR  QUESTIONS  OF  TIME  (Continued) 


bO-178B 

Reference 

Question 

Response 

Issues 

17. 

OIS 

What  was  the  total  duration  of  time  used  for 
this  computation?  Has  care  been  taken  to 
consider  only  the  appropriate  durations? 

18. 

12.3.5  k 

What  is  the  actual  error  rate  computed  after 
analyzing  the  service  history  data? 

19. 

12.3.5  k 

Is  this  error  rate  greater  than  the  proposed 
acceptable  error  rate  defined  in  PS  AC 
according  to  j.  (4)? 

20. 

12.3.5  k 

If  the  error  rate  is  greater,  was  analysis 
conducted  to  reassess  the  error  rates? 
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